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Abstract. We obtain a complete classification of the continuous unitary rep- 
resentations of oligomorphic permutation groups (those include the infinite 
permutation group Sao , the automorphism group of the countable dense linear 
order, the homeomorphism group of the Cantor space, etc.). Our main result 
is that all irreducible representations of such groups are obtained by induction 
from representations of finite quotients of open subgroups and moreover, every 
representation is a sum of irreducibles. As an application, we prove that many 
oligomorphic groups have property (T). We also show that the Gelfand-Raikov 
theorem holds for topological subgroups of Sao- for all such groups, continuous 
irreducible representations separate points in the group. 



1. Introduction 

Abstract harmonic analysis is classically restricted to studying representations 
of locally compact groups, and for a good reason: the Haar measure provides an 
invaluable tool for constructing and analyzing representations. It gives rise to 
the left-regular representation (so that every locally compact group has at least 
one faithful representation) but also allows to define the convolution algebra of 
the group and various useful topologies on function spaces which are important 
for understanding the representations. And indeed, many standard theorems of 
the subject break down for non-locally compact groups: for example, the group 
of orientation-preserving homeomorphisms of the reals has no non-trivial unitary 
representations whatsoever (Megrelishvili |M3| ). while the group of all measurable 
maps from [0, 1] to the circle has a faithful unitary representation by multiplication 
on L 2 ([0, 1]) but no irreducible representations (this example is due to Glasner |G2| : 
see also [BdlHVl Example C.5.10]). Those rather pathological examples suggest 
that any attempt to develop a representation theory for non-locally compact groups 
should be restricted to certain well-behaved classes. 

And indeed, a number of interesting classification results have been obtained for 
the representations of some concrete non-locally compact groups. Lieberman [L] 
classified the unitary dual of the infinite symmetric group Soo (for us, Soo is always 
the group of all permutations of the integers and not only those with finite sup- 
port). Later, Olshanski developed a rather versatile machine, called the semigroup 
method, using which he succeeded to give another proof of Lieberman's theorem |Q2) 
and also classify the representations of many other non-locally compact groups in- 
cluding the infinite-dimensional unitary, orthogonal, and symplectic groups, and 
(a variant of) the infinite-dimensional general linear groups over finite fields [01] 
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(cf. Remark 15.311 . See the survey |Ol| for an explanation of the method and the 
references therein for more examples. A classification for the representations of the 
unitary group had also been announced by Kirillov [K . All of the groups above have 
only countably many irreducible representations and every representation splits as 
a sum of irreducibles, a situation that very much resembles the case of compact 
groups. 

In this paper, we study the unitary representations of non-archimedean, separa- 
ble groups (i.e. separable groups that admit a countable basis at the identity con- 
sisting of open subgroups), also known as subgroups of Sao. This property alone 
allows us to recover one important result valid in the locally compact situation, 
namely the Gelfand-Raikov theorem. More precisely, we prove the following. 

Theorem 1.1. Let G be a topological subgroup of Soq. Then for every x,y G G, 
x 7^ y, there exists a continuous, irreducible, unitary representation n of G such 
that n(x) 7^ ir(y). 

The main body of the paper, however, concentrates on groups that have an ad- 
ditional special property, that of Roelcke precompactness (cf. Definition 12. ip . It 
turns out that this property has a natural translation in the language of permuta- 
tion groups and model theory. For us, a permutation group will be a topological 
subgroup of the group of all permutations of a countable set X. The following 
definition is of central importance for this paper. 

Definition 1.2. Let X be a countable (finite or infinite) set. A permutation group 
Gn-Xis called oligomorphic if the diagonal action G rx X™ has only finitely many 
orbits for each n. A topological group G is oligomorphic if it can be realized as an 
oligomorphic permutation group. 

Closed oligomorphic permutation groups also have a model-theoretic interpre- 
tation: they are exactly the automorphism groups of w-categorical structures (cf. 
Section [5]) . A standard way to produce w-categorical structures is the Fra'isse con- 
struction: given a class of finite structures satisfying a certain amalgamation prop- 
erty, there is a way to build a (unique) infinite, homogeneous structure that contains 
all structures in the class as substructures. We postpone the formal definitions to 
Section [2] and just describe a few examples. 
Examples of cj-categorical Fra'isse limits: 

(i) The Fra'isse limit of all finite sets without structure is a countably infinite 
set. The corresponding group is Sob, the group of all permutations of this 
set. 

(ii) The Fra'isse limit of all finite linear orders is the countable dense linear order 
without endpoints (Q, <). We denote the corresponding automorphism 
group by Aut(Q). 

(iii) The Fra'isse limit of all finite Boolean algebras is the countable atomless 
Boolean algebra which is isomorphic to the algebra of all clopen sub- 
sets of the Cantor space 2 N . The corresponding automorphism group is 
Homeo(2 N ), the group of all homeomorphisms of 2 N . 

(iv) The Fra'isse limit of all finite vector spaces over a fixed finite field ¥ q is the 
infinite-dimensional vector space over ¥ q . The automorphism group is the 
general linear group GL(oo,F g ). 

(v) The Fra'isse limit of all finite graphs is the random graph, the unique count- 
able graph R such that for every two finite disjoint sets of vertices U, V, 
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there exists a vertex x which is connected by an edge to all vertices in U 
and to no vertices in V. We denote its automorphism group by Aut(R). 
There are also many other cj-categorical structures, including, for example, cer- 
tain groups [HJ , and a variety of combinatorial structures [KPTJ . There is a rather 
extensive literature devoted to the subject; we refer the interested reader to the vol- 
ume [KM , or the more recent survey Macpherson |M2j and the references therein. 
We also indicate some ways to construct new oligomorphic groups from old ones in 
Proposition 12.21 

The main theorem describing the unitary representations of oligomorphic groups 
is the following. 

Theorem 1.3. Let G be an oligomorphic group. Then every irreducible unitary 
representation of G is of the form Indc(v')( cr )) where V < G is an open subgroup, 
C(V) is the commensurator ofV, V^iC(V), and a is an irreducible representation 
of the finite group C(V)/V. Moreover, every unitary representation of G is a sum 
of irreducibles. 

We also provide a criterion when two irreducible representations as above are 
isomorphic (Proposition 14. ip . 

As every oligomorphic group has only countably many distinct open subgroups 
(Corollary 12. 5j) . the theorem shows that every oligomorphic group has only count- 
ably many irreducible representations. 

Our methods are quite different from the approach of Olshanski. In particular, 
his semigroup method only applies when the group is obtained as the completion 
of an inductive limit of subgroups, which is not the case for many oligomorphic 
groups (for example, Aut(Q)). On the other hand, we have borrowed an important 
idea from Lieberman: the use of a weak limit point in the proof of Theorem 11.31 

If one is given a realization of a closed oligomorphic group as the automorphism 
group of a countable structure, it is possible to give a more concrete description of its 
representations in terms of the structure. For example, for the automorphism group 
of the random graph, all irreducible representations can be obtained in the following 
way. One takes a finite (induced) subgraph A C R and sets V to be the pointwise 
stabilizer of A. Then C(V) is the setwise stabilizer of A and C(V)/V = Aut(A). 
So in this case, the irreducible representations of Aut(R) are obtained by induction 
from irreducible representations of automorphism groups of finite graphs (and in 
fact, this correspondence is one-to-one if one makes the obvious identifications). 
See Section [5] for more details. 

As a corollary of the classification of the representations of Aut(Q), we obtain 
that the group Homeo + (R) has no non-trivial unitary representations (this is a 
special case of a result of Megrelishvili) , cf. Corollarv l5.5l 

As a further application, we establish property (T) for a large class of oligomor- 
phic groups. Our technique is quite similar to the one used by Bekka [B] to prove 
that the unitary group has property (T) (for which he used Kirillov and Olshanski's 
classification of its representations) . 

Theorem 1.4. All of the examples -(jvj) above have property (T). 

In all of those groups, it is also possible to find explicit finite Kazhdan sets. 
We also have a more general result (Theorem 16. 6 j) . which requires some additional 
terminology to state. The question whether all closed oligomorphic groups have 
property (T) remains open. 
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The paper is organized as follows. In Section[2j we recall the definition of Roelcke 
precompactness and provide a model-theoretic characterization of Roelcke precom- 
pact subgroups of Soo ■ In Section [3J we prove some basic results about represen- 
tations of non-archimedean groups, including Theorem ll.il In Section 21 we prove 
the main theorem. Section [5] is devoted to some model-theoretic considerations and 
calculations in specific examples. In Section [6j we discuss property (T). 

Notation. If G is a group and g,x £ G, g x denotes the conjugate xgx^ 1 . Note 
that this is the conjugation action on the left, so that (g x ) y = g yx ■ If G is the 
automorphism group of a structure X and a £ X™ is a tuple, Gs, denotes the 
stabilizer of all elements of a. If A C X is a substructure, Ga is the setwise 
stabilizer of A (the set of all g such that g ■ A = A) and G(a) is the pointwise 
stabilizer (the set of all g such that g ■ a = a for all a £ A). If X is a set, X^ 
denotes the set of all subsets of X of size n. 

A representation of a group G is always a unitary representation. If 7r is a 
representation, H(tt ) denotes its Hilbert space. All Hilbert spaces are complex. 

Acknowledgements. I would like to thank Ita'i Ben Yaacov and C. Ward Henson 
for an important insight for the proof of Lemma 12.61 Martin Hils for explaining 
to me some basic model theory, and H. Dugald Macpherson for pointing out an 
error in a preliminary version of the paper. I am grateful to the Fields Institute in 
Toronto for its hospitality while this paper was being finished. I am also grateful 
to the referee for a number of helpful comments. 

2. Oligomorphic groups and Roelcke precompactness 

2.1. Roelcke precompact topological groups. A topological group G is pre- 
compact iff the completion of its left uniformity is compact iff the completion of 
its right uniformity is compact. In that case, the common completion of the left 
and the right uniformities has the structure of a compact group in which G embeds 
topologically as a dense subgroup. In particular, if G is Polish, G is precompact iff 
it is compact. The condition that the left uniformity on a group G is precompact 
can be written as follows: for every neighborhood U of the identity, there exists a 
finite set F C G such that FU = G. 

There exists a weaker notion of precompactness that will be central for this 
paper. A topological group is called Roelcke precompact if the completion of its 
lower uniformity (the greatest lower bound of the left and right uniformities) is 
compact. We will find it, however, more convenient to work with the following 
direct definition. 

Definition 2.1. A topological group G is Roelcke precompact if for every neigh- 
borhood of the identity U, there exists a finite set F C G such that G = UFU . 

The notion of Roelcke precompactness is weak enough to include many inter- 
esting non-compact examples but still sufficiently powerful to allow the generali- 
sation of some results valid for compact groups. Roelcke precompact groups were 
introduced by Roelcke-Dierolf [RD], who also gave the first examples, and were 
later studied by many authors, including Uspenski and Glasner. Known examples 
of Roelcke precompact groups include the unitary group of a separable, infinite- 
dimensional Hilbert space (Uspenski |U2| ). the isometry group of the bounded 
Urysohn metric space (Uspenski |U4j ). and the automorphism group of a standard 
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probability space (Glasner |G1| ). We also note that Roelcke precompactness is only 
interesting for "infinite-dimensional" groups: a locally compact group is Roelcke pre- 
compact iff it is compact (to see this, note that if U is a compact neighborhood of 
the identity, then G = UFU is compact). We suggest the survey |U3) by Uspenski 
as a general reference. 

We start by showing that the class of Roelcke precompact groups enjoys most of 
the closure properties of the class of compact groups with the notable exception of 
closure under taking closed subgroups. (In fact, as was shown in |U4| . every Polish 
group embeds as a closed subgroup of a Polish Roelcke precompact group.) 

Proposition 2.2. The following statements hold: 

(i) If it: G — > H is a continuous homomorphism with a dense image and G is 
Roelcke precompact, then so is H . 

(ii) If G± and G2 are Roelcke precompact, then so is Gi x G 2 . 

(iii) The inverse limit of an inverse system of Roelcke precompact groups is Roel- 
cke precompact. In particular, an arbitrary product of Roelcke precompact 
groups is Roelcke precompact. 

(iv) // G is Roelcke precompact and H < G is open, then H is Roelcke precom- 
pact. 

(v) If N is a normal subgroup of G such that both N and G/N are Roelcke 
precompact, then so is G. 

Proof. (PJ. Let U C H be an open neighborhood of 1 . Find an open V such that 
1 G V 2 C U. Let F C G be finite such that G = n- 1 (V)F'K- 1 {V). We check that 
Uir(F)U = H. Let h G H . By the density of n(G) in H, there exists g G G such 
that w(g) G hV~ l . Then g — Ui/i>2 for some t>i,«2 G 7r — 1 (V) and / G F. Finally, 
h G 7r(g)V = w(vi)ir(f)<ir(v 2 )V C Un(F)U. 

([H|). If Ui x U2 is an open neighborhood of the identity in G\ x G2 and F\ C Gi, 
F 2 C G 2 are finite such that UiF x Ui = d, U 2 F 2 U 2 = G 2 , then Gi x G 2 = 
(C/j x U 2 ){F 1 x F 2 )(Ui x U 2 ). 

([mil . Let G = hmi/i. Let [/ = ir^ l (Ui) be an open neighborhood of 1 in G, 
where C/j is an open neighborhood of 1 in H t . Then there exists a finite F C Hi 
with i?. t = f/jFUi. If F' C G is finite with tt^F') = F, we check that [/F'C/ 2 = G. 
Indeed, let # in G. Then there exist ui, 112 G G, / G F' such that 7r(x) = Tr(uifu 2 ). 
Then there is ft. G ker^ such that x = U\fu2h G UF'U 2 (because ker^ C {/ by 
the definition of U). 

(|rv|) . Let [/ C be an open neighborhood of 1. Then J7 is open in G and 
there exists a finite F C G such that G = UFU. Now one easily checks that 
F = U(FHH)U. 

Ijyj). Let C/ C G be an open neighborhood of 1. Let Q = G/7V and denote by 
7r: G — > Q the quotient map. Then 7r(J7) is open in Q. Find a finite F C G such 
that 7r(C/)7r(F)7r([7) = Q and assume also that 1 G F. Let V = N n f|/ ei r C // and 
note that V is relatively open in N. Find a finite B C N such that VFI^ = iV. We 
claim that U 2 BFU 2 = G. Indeed, fix x G G. Find / G F and ui, u 2 G J7 such that 
7t(mi/u 2 ) = 7r(a;). Then there exists h £ N with a; = hu\fu2- Find fi,w 2 G and 
b E B such that /itti = «i6u 2 . Finally, we have 

a; = huifu2 = u\{u^ 1 hui)fu2 — uiDi&z; 2 /m 2 
= u 1 v 1 bf{f- 1 V2f)u 2 G U 2 BFU 2 , 
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finishing the proof. □ 

We end this subsection with a simple application. In [R2], Rosendal introduced 
property (OB), intended as a generalization of the well known properties (FA) and 
(FH). A topological group has property (OB) if every time it acts (separately) 
continuously by isometries on a metric space, every orbit is bounded. 

Proposition 2.3. Every Roelcke precompact group has property (OB). 

Proof. Let G be a Roelcke precompact group and G rx (X, d) a separately contin- 
uous action on a metric space by isometries. For every point xq G X, the function 
g h- > d(xo,g ■ xq) is uniformly continuous in the lower uniformity, as can be seen 
from the inequality 

d(x , high 2 ■ so) = dih^ 1 ■ xq, gh 2 ■ x ) 

< dih^ 1 ■ xq, xq) + d(x 0l g ■ Xq) + d(g ■ xq, gh 2 ■ x ) 

= dih^ 1 ■ xq, x ) + d(x 0l g ■ Xo) + d(x , h 2 ■ x ). 

It thus extends to the compact completion and must be bounded. □ 

2.2. Permutation groups and closed subgroups of Soo- We now concentrate 
on the main objects of study in this paper, namely infinite permutation groups. 
Let Sea be the group of all permutations of a countable infinite set X. It becomes 
naturally a topological group if equipped with the pointwise convergence topology, 
where X is taken to be discrete. A permutation group is a topological subgroup of 
the group of all permutations of X. 

It is well known that the topological groups that can be realized as permutation 
groups are exactly the separable topological groups that admit a countable basis 
at the identity consisting of open subgroups (those groups are often called non- 
archimedean) . The basis of open subgroups is given by the pointwise stabilizers of 
finite subsets of X. 

A natural way in which closed permutation groups arise in practice is as automor- 
phism groups of countable structures in model theory, for example, automorphism 
groups of countable graphs, countable orders, or various algebraic structures. Of 
special interest to us will be the oligomorphic groups (see Definition II .2[) for which 
it is possible to translate back and forth between model-theoretic and permutation 
group-theoretic language. For a gentle introduction to the subject of oligomorphic 
groups, we refer the reader to Cameron [CI] . It turns out that there is a close 
connection between the properties of being Roelcke precompact and oligomorphic. 
To see this, we first reformulate Definition 12 . 1 1 for non-archimedean groups: a topo- 
logical subgroup G of Soo is Roelcke precompact iff for every open subgroup V < G, 
the set of double cosets 

V\G/V = {VxV : x € G} 

is finite. 

Theorem 2.4. For a topological subgroup G < Soo, the following are equivalent: 

(i) G is Roelcke precompact; 

(ii) for every continuous action G r\ X on a countable, discrete set X with 
finitely many orbits, the induced action G r\- X™ has finitely many orbits 
for each n; 

(iii) G can be written as an inverse limit of oligomorphic groups. 
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Proof. =>■ ([li]). Without loss of generality, we can suppose that the action G rv X 
is transitive. We use induction on n. The case n = 1 is given by the hypothesis. 
For the induction step n — > n + 1, it suffices to find for every a £ X™ , a finite set 
-B(a) C X such that for every d g X, there is ft e G a and 6 € -6(a) such that 
d = h ■ b. Then if {d\, . . . , a s } is a complete set of representatives for the orbits of 
G rx X™, {(di,b) : i < s,b £ 5(5^)} will be a complete set of representatives for 
the action G rx X" +1 . Indeed, let (c,d) £ X n+1 . Using the induction hypothesis, 
find g £ G such that g-di — c. Find h £ G Bi and b £ B(di) such that g^ 1 ■ d = h-b. 
Then one has 

gh ■ (di,b) = g ■ (a^h ■ b) = (c, d). 

Fix now d £ X™, let Co be an arbitrary element of X, and let {Gac goGac , ■ ■ ■ , 
Gac 9kGac } be a complete list of the double cosets of G S c - Set B{a) — {gi ■ cq : 
i = 1, . . . , k}. Let now d £ X be arbitrary, d — g ■ cq (using the transitivity of the 
action). Let i, hi, hi be such that hi, hi £ Gac and g = h\gihi. We have 

d = 9 ■ cq = h^gihi ■ c = h x g t ■ c , 

finishing the proof. 

(pi]) =>- (jliT]) . Let {V n : n £ N} be a basis at \q of open subgroups such that 
for all n, V n +i < y„. Then for each n, G acts continuously by permutations on 
the discrete set G/V n and this gives rise to a continuous homomorphism 7r„ : G — > 
Sym(G/Vn)- The groups {7r„(G) :n£N} form a directed system (there are natural 
maps 7r„_|_i(G) — > 7r„(G)) and its inverse limit is isomorphic to G. Each 7r„(G) is 
oligomorphic by the hypothesis. 

(fm)) (0). In view of Proposition !!??!?! it suffices to show that if G is oligomorphic, 
then it is Roelcke precompact. Let G be a group of permutations of X such that 
the action G rx X is oligomorphic. It suffices to show that every stabilizer Gg, for 
d £ X" has finitely many double cosets. If {(a, g\ ■ a), . . . , (a, g^ ■ a)} is a complete 
list of representatives for the G-orbits on X 2 " that are subsets of G • a x G • a, then it 
is easy to check that {G a giG a : i = 1, . . . ,k} exhausts all double cosets of G a - □ 

Remark. A proof that every (even approximately) oligomorphic group is Roelcke 
precompact is essentially contained in |R2| . Also, some special cases of the above 
theorem had been known before: Uspenski had shown that and Homeo(2 N ) are 
Roelcke precompact |U1UU3| . 

A basic application of the theorem is the following corollary which had been 
noted before by many authors. 

Corollary 2.5. Every oligomorphic group has only countably many distinct open 
subgroups. 

Proof. Fix a countable basis at the identity of open subgroups {Vi}. Every open 
subgroup contains a basic open subgroup and is therefore a union of finitely many 
double cosets of some Vi. □ 

Now we turn to some basic group-theoretic lemmas about oligomorphic groups 
that will be used later. Recall that two subgroups of a group are commensurate 
if their intersection has finite index in both. If H < G is a subgroup, define the 
commensurator of H in G to be 

ComuiG(H) — {g £ G : H and H 9 are commensurate}. 
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It is a standard fact that CommG(-ff) is a subgroup of G containing H . If H\ and 
H2 < G are commensurate, then CommG(i?i) = CommG(-ff2)- Note also that the 
number of left cosets in HgH is equal to [H : H n H 9 ] and the number of right 
cosets is [H 9 : Hl~)H 9 ]. The following lemma will be particularly useful for studying 
commensurators in oligomorphic groups. I am grateful for the idea of the proof to 
Ita'i Ben Yaacov and C. Ward Henson. 

Lemma 2.6. Let G be a Roelcke precompact group and V < G an open subgroup. 
Then, for every x G G, the double coset VxV contains finitely many left cosets of 
V iff it contains finitely many right cosets of V . 

Proof. Note first that if Hi, H2,Hz < G and H1H2 contains finitely many f/2-cosets 
and H2H3 contains finitely many i?3-cosets, then H1H3 C H1H2H3 also contains 
only finitely many iJ3-cosets. 
For H < G, denote 

J-(H) = {yH : HyH contains finitely many left i?-cosets} 

= {yH : HH y contains finitely many i? y -cosets}. 

Note that T(H) is exactly the union of double cosets that contain only finitely 
many left cosets and as G has only finitely many double cosets of V, J-(V) is finite. 
Now we show that 



Indeed, we have that VV X contains finitely many V x cosets and VV y contains 
finitely many V y cosets. Conjugating the latter by x, we obtain that V x V xy con- 
tains finitely many V xy cosets. Now applying the observation in the beginning of 
the proof, we have that VV xy contains finitely many V xy cosets, i.e. xyV G ^"(V). 

Suppose now that x G G is such that xV G J~(V) and consider the map 
$: G/V -> G/V defined by <S>(yV) = xyV. By flU]), $(-F(V)) C T(V) and as 
F(y) is finite and $ is injective, in fact, $(J 7 (V r )) = .F(V). Therefore there ex- 
ists yV G ^(V) such that $>(yV) = V, i.e. yV — x~ x V . So we conclude that 
x~ x V G J-(V) or equivalently, that VxV contains finitely many right cosets. The 
other direction of the statement is obtained by replacing x with x~ x . □ 

As a corollary, we obtain that we can define the commensurator of an open 
subgroup in a Roelcke precompact group as 

Comm<3(V) = M{^5^ : V g V contains finitely many left cosets of V} 



Lemma 2.7. Let G be Roelcke precompact and V < G be open. Then the following 
hold: 

(i) [Comm G (V) : V] < 00; 

(ii) CommG(CommG(F)) = Commc^); 

(iii) IfV\,V2 < G are open and ComniG(Vi) and CommG(V2) are commensu- 
rate, then Commc(l/i) = Commc^). 

Proof. (JTJ) . Since V < Commc^), V is open, and G is Roelcke precompact, 
ComniG(^) is a union of finitely many double cosets of V. By (|2.2j) . every dou- 
ble coset VxV in CommG(V) contains only finitely many left cosets, so the claim 
follows. 



(2.1) 




(2.2) 




: VgV contains finitely many right cosets of V}. 
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(JTTJ) . Follows from (JXJ> . 
fm|. We have 

ComrriG^Vi) = Comm^ComnxG^)) = Comm^CommG^T^)) = Commc^V^). 

□ 

We see that Commcfy) is the maximal subgroup of G containing V in which 
V is of finite index. In view of (JTTJ) , we say that an open subgroup H of a Roelcke 
precompact, non-archimedean group G is a commensurator if Commc(Jf) = H. 

2.3. Examples from model theory. A natural class of permutation groups is 
obtained from model theory. A signature (or a language) £ is a collection of relation 
and function symbols, where each symbol has a certain arity n(-). A structure 
for £ is a set X together with interpretations for the symbols: for each relation 
symbol R : a relation i? x C X n ( fl ' and for each function symbol F, a function 
F x : ~X. n ( F > — > X. An automorphism of the structure X is a permutation that 
preserves the relations and the functions. A structure is relational if the signature 
has no functions symbols. 

Every closed permutation group can be obtained as the automorphism group of 
a relational structure: one just adds a relation for every orbit on X™ for all n (see, 
for example, |BK] for more details). An important model-theoretic characterization 
of the structures whose automorphism groups are oligomorphic is given by the 
following classical theorem (see (Hj for a proof). 

Theorem 2.8 (Ryll-Nardzewski, Engeler, Svenonius). For a countable structure 
X, the following are equivalent: 

• X is lo- categorical; 

• Aut(X) rx~X. is an oligomorphic permutation group. 

A structure X is called ui- categorical if X is the unique (up to isomorphism) 
countable model of the first-order theory of X. 

An especially attractive situation is when X is homogeneous in the following 
strong sense: every isomorphism between finite substructures of X extends to a 
full automorphism of X (sometimes those structures are called ultrahomogeneous) . 
A classical theorem of Fraisse describes the homogeneous structures as the Fraisse 
limits of classes of finite structures satisfying a certain amalgamation property 
[Hi Section 7.4]. A homogeneous structure X is cj-categorical iff for every n there 
are only finitely many isomorphism types of substructures of X generated by n 
elements. In particular, every homogeneous structure in a finite, relational signature 
is oj-categorical. We also see that all examples given in the introduction are w- 
categorical. We add one slightly less known, but instructive, example to the list. 
Examples (cont.): 

(vi) (Cherlin and Hrushovski) Consider a signature with infinitely many relation 
symbols {E n } n >i, where E n is of arity 2n. Let X be the class of all finite 
structures in this signature, where each E n is interpreted as an equivalence 
relation on subsets of the structure of size n with at most 2 equivalence 
classes. For every k £ N, there are only finitely many structures in X of size 
k, so the Fraisse limit X of X is cj-categorical. What is remarkable about 
this structure is that its automorphism group G has a quotient isomorphic 
to (Z/2Z) N . The reason for this is that G acts on the sets X^/E n all 



10 



TODOR TSANKOV 



of which have size 2. Extending this example, Evans and Hewitt |EH] 
constructed for every profinite, metrizable group H an oligomorphic group 
G such that if is a quotient of G. 

Typical homogeneous structures that are often encountered in the literature 
which are not w-categorical are the discrete structures that arise as approximations 
of continuous ones: the rational Urysohn metric space, countable measured Boolean 
algebras, etc. 

3. Representations of permutation groups 

A unitary representation of a topological group G is a strongly continuous ho- 
momorphism n: G — > U(H) to the unitary group of some Hilbert space H, (i.e. the 
map G — > "H, g > 7r(<7)£ is continuous for every £ G %). A permutation group 
G rx X has some natural representations defined using the action on X, namely, 
G rx f 2 (X n ) for n € N. Those representations clearly separate the points of G. 
In this section, we show that irreducible representations also separate points. (A 
representation 7r is irreducible if does not have non-trivial subspaces invariant 
under tt.) This can be considered as a version of the classical Gelfand-Raikov |GR| 
theorem for subgroups of Soo . 

If 7r is a representation of G and V < G, denote by 'Hy(7r) the closed subspace 
of fixed points of V. We start with a simple but key lemma a version of which had 
been previously used by Lieberman [L] and Glasner- Weiss [GW . 

Lemma 3.1. Let G be a subgroup of and {Vi : i € N} be a basis at the identity 
for G consisting of open subgroups. Then for any continuous unitary representation 
tt of G, the space Ui'Hvii'x) is dense in H(tt). 

Proof. Let £o G H(tt) be an arbitrary vector and fix e > 0. As the representation 
is continuous, there is i € N such that 7r(Vi)£o is contained in the ball with radius 
e around £q- Let C be the closure of the convex hull of 7r(Vi)£o an d let n be the 
unique element of least norm in C. As ir(Vi)C — C and 7r preserves the norm, n is 
a fixed point of ir(Vi). Also, by the choice of Vi, — £o|| < e - O 

Recall that a continuous function <fi : G — > C is called positive definite if for every 
xi, . . . ,x n € G and a, . . . , c„ € C, 

n 

(3.1) <p(x~ 1 x i )c i cj > 0, 

i,j=l 

i.e. the matrix {(^{xj 1 Xi)) t . is positive-definite. If 7r is a representation of G and 
£ £ H(it)i the function x i— > (7r(x)£, £) is positive definite and conversely, the GNS 
construction produces from a positive definite function </> a representation 7r and a 
cyclic vector £ 6 (i-e. such that the linear span of the orbit 7r(G)£ is dense in 

H(tt)) such that 

(j)(x) = (7r(x)£, £) for all x G G 

(see |BdlHV[ Appendix C] for more details). In particular, \4>(x)\ < 0(1) for all 
x G G. We now have the following basic observation. 

Lemma 3.2. Let G be a group and H < G a subgroup. If (p is a positive definite 
function that is constant on H, then it is constant on double cosets of H . 
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Proof. Let tt and £ £ 'H( 7T ) be such that 4>{x) — (tt(x)^,^) for x £ G. Then by the 
hypothesis, for any h £ H, (w(h)^, £) = (7r(l)£, £) = ||£| , i.e. £ G T-Lh{^)- Now we 
have for any x £ G and h\,h2 £ H: 

<j>{h x xh 2 ) = (Tr(h lX h 2 %0 = (irixWhi^irih^O = (ir(x)S,Q = 0(z), 

finishing the proof. □ 

Let 

Vi{G) ={(/>'■ G —> C : is positive definite and 0(1) = 1} 
and if V < G is open, let also 

T V (G) = {0 G -Pi(G) : 0(u) = 1 for all w € 7}. 

By Lemma l3~2l we can consider Vv{G) as a subset of £°°(V\G/V). Vv(G) is convex 
and bounded and by the definition (|3.ip . it is also closed in the weak* topology of 
£°°(V\G/V) and thus compact. 

Lemma 3.3. If <fi is an extreme point ofVy{G), then it is also an extreme point 
ofV x {G). 

Proof. Suppose that </> £ Vv(G) and ipi,ip2 £ V\(G) and t £ (0,1) are such that 
= ttpx + (1 — t)V>2' For every v £ V, we have 

1 = <j>( v ) = *ReVi(«) + (1 - t)Re ^(») < tyi(l) + (1 - 0^(1) = L 

showing that we must have equality in the middle, i.e. Re"0i(v) = Re02(^) = 1 
for all v £ V . As |^i(t;)|, IV^v)! < 1, this implies that ipi (v) — ip2(v) — 1. Thus 
"01 j "02 £ 7V(G), proving the lemma. □ 

We now see that the classical proof of Gelfand-Raikov extends to our situation. 

Proof of Theorem \l.l\ It suffices to show that for every x £ G, x =/= I, there is an 
irreducible representation n such that n(x) 7^ 7r(l). Recall that a representation 
7r with a cyclic unit vector £ is irreducible iff the corresponding positive definite 
function is an extreme point of Vi(G) [BdlHV, Theorem C.5.2]. Let now 1 7^ x £ 
G. Let V < G be an open subgroup such that x ^ V. Consider the positive 
definite function xv (the characteristic function of V) which corresponds to the 
representation G rx £ 2 (G/V) with cyclic vector Sv- We have that xv £ Vv(G) 
and xv (x) 7^ xy(l). Consider now the weak* closed, convex set C — {0 £ Vv{G) : 
4>{x) — 0(1)}. As C C Vv(G), by the Krein-Milman theorem, there exists an 
extreme point of 7V(G) such that ^ G. By Lemma \'S. 31 is also an extreme 
point of V\ (G), producing the required irreducible representation. □ 

4. Representations of oligomorphic groups 

Let G be a subgroup of S'oo and G rx Y be a continuous action on a discrete, 
countable set Y. There is a natural associated representation of G on £ 2 {Y) and if 
Y = \_\j Yi is the decomposition of Y into orbits, we have that £ 2 (Y) — (Bi^ 2 (^)- 
Therefore we can as well suppose that the action G rx Y is transitive; in this case, 
the corresponding representation is just the quasi-regular representation £ 2 (G/V) 
for some open subgroup V of G. In order to describe those, we recall the notion of 
induced representation. 

Let G be a topological group and H be an open subgroup of G. Let a be a 
representation of H. The induced representation Ind^(cr) is defined as follows. Let 
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T be a complete system of left coset representatives of H in G. Let M be the space 
of all functions /: G — > H(a) for which 

(4.1) f(gh) = aQi-^fig) for all g eG,heH. 

In particular, for / 6 M, ||/(a;)|| is constant on left cosets of H. For / £ M, define 

(4-2) 11/11 = (Ell/(-9)H 2 ) 1/2 

and note that because of the above observation, ||/|| does not depend on the choice 
of T. Let H = {/ 6 M : ||/|| < oo}. Then the representation Ind#(cr) on the 
Hilbert space % is defined by 

(lnd G H {a){g)-f){x) = f{g- 1 x). 

As H is open, the representation Ind^r(er) is continuous. For example, the quasi- 
regular representation £ 2 (G/H) can be written as Ind^(lff), where 1h is the trivial 
one-dimensional representation of H. 

We note that as we only need to induce from open subgroups H < G, the 
homogeneous space G/H always carries the counting measure and we are spared 
the measure-theoretic complications that occur in the locally compact setting. For 
more details on induced representations, see, for example, [BdlHV, Appendix E]. 

Suppose now that G is Roelcke precompact and fix an open subgroup V < G. 
Let H be a subgroup of G such that V < H. As for normal subgroups double 
cosets coincide with left cosets, V has finite index in H. Denote by K the finite 
group H/V and by Xk the left-regular representation of K, which we will also 
consider as a representation of H. Then using the theorem about induction in 
stages ( [BdlHVl Theorem E.2.4]), we have 

(4.3) £ 2 (G/V) S Ind«(l y ) = Indg ( Ind^(ly)) ~ Indg(A^). 

As Xk splits as a sum of irreducible representations of K (and in fact all irreducible 
representations of K occur as direct summands), we are led to consider representa- 
tions of the form Ind^r(er), where H is an open subgroup of G and a is an irreducible 
representation of some finite quotient of H. There is a general criterion known as 
the Mackey irreducibility criterion for determining whether representations of the 
form Ind^r(cr) are irreducible for H an open subgroup of G and a an irreducible 
finite- dimensional representation of H. The criterion is usually stated for discrete 
groups but works equally well in this more general setting. It is due to Mackey |M1| 
when a is one-dimensional and to Corwin |C3| in the general case. Below we state 
and prove a special version of the criterion adapted to our situation. 

If H < G, g € G, and a is a representation of H, define the representation a 9 of 
H3 by 

<T 9 (x) = CT^ 1 ). 

Proposition 4.1. Let G be a Roelcke precompact subgroup of Soq. Then the fol- 
lowing hold: 

(i) If H < G is a commensurator, V <! if is open, and a is a representation of 
H/V, then lnd H (a) is irreducible iff a is. 

(ii) // Hi , H2 < G are commensurator s, V\ < H\ , V2 <3 Hi are open, and 
o~i, 02 are irreducible representations of H\jV\, H2/V2, respectively, then 



UNITARY REPRESENTATIONS OF OLIGOMORPHIC GROUPS 



13 



Indfl- (<7i) = Ind ff2 (CT2) iff there exists g € G such that Hi = iff and 
cr 2 = <J\ . 

Proof. 0. If a = 01 © (T 2 , then Indg(a) = Indg(<7i) © Indg(cr 2 ). 

(<£=) Suppose cr is irreducible and denote 7r = Ind^(cr). We first show that 

(4.4) H v (k) = {/ € H(ir) : /(z) = for x £ if}. 

Suppose first that f(x) = for all x £ if. Let g G V. If x £ if, then g~ x x £ if 
and = f(x) = fig^x) = (ir(g)f)(x). If x € if, then 

(n(jg)f)(x) = f^x) = fixx^g^x) = a(x- 1 gx)f(x) = f{x) 

as a is trivial on V . For the other direction, suppose that / G "H(7r) is ^-invariant 
and a; G G is such that /(x) 7^ 0. If x £ H, then as if is its own commensurator, 
by (|2.2[) . HxH contains infinitely many left cosets of if. Since [if : V] < 00, Vrrff 
also contains infinitely many left cosets of if. As / is ^-invariant, its norm must 
be infinite, contradiction. We thus obtain that 

(4.5) / 1 y /(f) is an isomorphism between tt(H)\u v (tt) and cr. 

Now suppose that it is reducible, i.e. H(tt) = JCQ3JC 1 - , where JC is 7r(G)-invariant. 
As the projection onto K commutes with n(V), we have 

H v (tt) = (Hv(k) n/C) © CHy(vr) n /C^) 

and the two parts on the right-hand side are 7r(ff)-invariant. By (|4.5I1 and the 
irreducibility of a, either Hv(ir) C /C or Hv(tt) C /C x . Since by (|4.4|l . Wv(tt) is 
cyclic for 7r, we have that £ = 'H(tt) or /C -1- = H(tv), proving that tt is irreducible. 

(p!]). (4=) Let T: H(<ti) —> H(a?) be a unitary operator that realizes the equiv- 
alence erf = o~2 (i.e. o~iT — Taf). Let iti — Ind^. (crj) and define the map 
U: H(tti) -> W(7r 2 ) by 

[/(/)(x)=T/(^ 1 ) 

It is not difficult to check that [/ is a well-defined unitary equivalence between m 
and 7T2. 

(=>) Suppose that 7Ti and -k-i are equivalent. Then there exists a non-zero / G 
T-L(tti) which is invariant under 711 (V^)- By the same argument as in we obtain 
that there is g G G such that VigH\ contains only finitely many left cosets of Hi, 
or, equivalently, [V2 : Vi H iff] < 00. Symmetrically, we find h G G such that 
[Vi : Vi n ff 2 ] < 00. For two subgroups A, B < G, say that A is large in B if 
[B : A n £?] < 00. As [if; : Vi] < 00, we have that 

(4.6) Hf is large in H2, and 

(4.7) H$ is large in Hi, 

so by conjugating (|4.6[) by /i and using transitivity, we can conclude that H\ 9 is 
large in H±. Applying Lemma 12.61 we obtain that Hi and H^ 9 are commensurate 
and therefore equal (by Lemma [2~71 (|m|) ) . Conjugating (|4.7p by g~ x hr x , we obtain 

that iff is large in iff ,l = Hi, while conjugating (|4.6p by we see that 

Hi is large in iff , so that ifi and iff are commensurate and therefore equal. 
So finally, if 2 = iff. 

Now let n'i = Ind^s (erf) = Ind^ 2 (erf). By the direction and the hypothesis, 
tt'i = Hi = 7T2. Let [/: - H(7r 1 ) — > %{'K2) realize the equivalence. Then we must 
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have U(Hv 2 ( 7r i)) = Hv 3 {^2)- By f|4.5j) and the fact that U commutes with the 
-fraction, we have that a% = erf . □ 

We are now ready to prove the main theorem. 

Theorem 4.2. Suppose that G is a Roelcke precompact subgroup of Sao- Then 
every unitary representation of G is a sum of irreducible representations of the 
form Ind^r (or), where H is a commensurator and a is an irreducible representation 
of H that factors through a finite quotient of H . 

Proof. Let ir be a representation of G. For £ e H(tt), let cj)^(x) = (ir(x)£,£) be the 
positive definite function on G associated to £. If £ is fixed by an open subgroup 
Vq < G, then by Lemma l3~2l is constant on double cosets of Vo, so the function 
(j)^ takes only finitely many values. By Lemma 13.11 there exists some non-zero 
£ £ which is fixed by an open subgroup. Choose now a non-zero £0 G ^( 7r ) 

such that <j)^ g takes the minimum possible number of distinct values. Let 

V = {geG: n(g)Z = £0} = {.9 € G : fc (ff) = ||£o|| 2 } 
and note that as the image of <j)^ is discrete, V is open. 
Claim. If g ^ Commc(V), then 4>£ (g) = 0. 

Proof. Let g ^ Coming (V) be arbitrary. By (|2.2p , contains infinitely many 

left cosets of V. Towards a contradiction, suppose that (ir{g)£o, £0) 7^ 0. Let 
higV, h,2gV, ... be distinct left cosets of V with hi E V for all i. Set £j = n(hig)^o 
and let 77 be a weak limit point of the £,s. By passing to a subsequence, we can 
assume that — > tt ' 77. Since 

(6,£o) = (7r(^ff)£o,£o) = (7r(p)Co,7r(^ -1 )^o) = (""(flOCo, £0) 

is bounded away from 0, we have that 77 7^ 0. Next we observe that the set of values 
of (f)^ is a subset of the set of values of 4>^ . Indeed, fix i £ N and note that for all 
x € G, we have 

(7r(a:)?7,6) = (»7, 7r(a: -1 )&) 

= lim (ir(h : jg)£ Q ,ir(x~ 1 h l g)£ ) 

= lim <t>e, a {g~ l K Xxh i9)- 

Now, taking limits as i — > 00, 

0») 0*0 = ( 7r ( x )v,v) = I™ (ir{x)r],£.i) = 1™ lim cj) (o (g~ 1 h~ 1 xhjg). 

I— J-OO Z— >00 J — ► OO 

As the image of ^ is discrete, 4> n (x) is a value of 

On the other hand, note that ||?7|| < ||£o||- Indeed, if H77I = ||£o| = \\Ci\\> then 
£i converges to 77 in norm, so for all e > 0, there exists N such that for i,j > N, 
ll£t — £7 II < e - As HCi — ca n take only finitely many values, the sequence £3 
is eventually constant. This contradicts the assumption that hig and hjg are in 
different left cosets of V . It follows that the set of values of <p v is a strict subset 
of the set of values of (as ||£o|| 2 is a value of (f>^ which is not a value of 4> v ), 
contradicting the choice of £0 . This completes the proof of the claim. □ 
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Put now H = Comm G (y) and V = f] heH V h . Then V'<H and as V has finite 
index in H, V' also has finite index in H. Let K, = span{7r(g)^o : g G H} and note 
that K, is finite-dimensional. 

Claim. n{x)K _L n{y)K if xH ^ yH. 

Proof. Let g,h £ H . We have: 

(n(xg)£ ,n(yh)£o) = {^{h^y^xg^o^o) = (f>^(h^ 1 y^ 1 xg) = 0. 

The last equality follows from the fact that if y~ 1 x ^ H, then h~ 1 y~ 1 xg ^ H and 
the previous claim. □ 

Note now that JC is fixed pointwise by V'. Indeed, if g £ H and h £ V', by 
the definition of V', hgV = gV, so there exists h' £ V such that hg = gh' and 
7r(/i)7r(g)£o = Tr(gh')£o = n(g)^Q. Thus we obtain a representation of H on K. that 
factors through H/V'. Denote this representation by a. Let T be a system of left 
coset representatives for H. We verify that the partial isometry U : H(lnd ( ^(a)) — > 
H(tv) given by 

U{f) = ® xeT *{x)f{x) 

does not depend on the choice of T and is a unitary equivalence between Ind^(cr) 
and the cyclic subrepresentation of tt generated by £o- That it does not depend on 
T follows from (|4.1[k to check that it intertwines the representations, observe that 

U{lnd G H {a){g)f)(x) = aeT n^fig^x) 

= 7%) ([/(/)(*)). 

So we obtained that it contains a subrepresentation that is isomorphic to Ind^(<7), 
where a factors through a finite quotient of H. By passing to a subrepresentation 
if necessary, we can assume that a is irreducible. Then Ind^(cr) is irreducible by 
Proposition 14.11 Now using Zorn's lemma, we conclude that 7r is actually a sum of 
such representations. □ 

5. Open subgroups and imaginaries 

In the previous section, we saw that in order to describe completely the repre- 
sentations of an automorphism group of an w-categorical structure, it suffices to 
understand the lattice of its open subgroups. It is most natural to understand the 
open subgroups in terms of the structure the group acts on. As we will see below 
(and as is well known), the open subgroups of the automorphism group correspond 
precisely to the imaginary elements of the structure. A particularly simple situation 
is when we can see all the open subgroups already in the structure itself, that is, 
when the structure eliminates imaginaries in a suitable weak sense. We proceed 
now with the formal definitions. 

Let X be an w-categorical structure and G = Aut(X). Recall that by the Ryll- 
Nardzewski theorem, a set A C X™ is (first-order) definable iff it is G-invariant. 
We are going to use the two terms interchangeably. A tuple b £ X™ is algebraic 
over a £ X m if the orbit G s • b is finite. The algebraic closure of a is the set of all 
b £ X algebraic over a. As X is w-categorical, the algebraic closure of a finite set is 
always finite. An imaginary element of X (or just an imaginary) is an equivalence 
class of some definable equivalence relation on a definable subset of X™. If 9 is a 



16 



TODOR TSANKOV 



definable equivalence relation and a <E X™, a/ 8 will denote the ^-equivalence class 
of a. If a — a/9 is an imaginary, we will denote by G a the stabilizer of a in G. As 
Go, < G a , G a is open in G. Conversely, if V < G is an open subgroup, there exists 
a such that G a < V. If we define the equivalence relation 9 on G • a by 



then is G-invariant and V — G a /e- 

Recall that the structure X admits weak elimination of imaginaries if for every 
imaginary a, there exists a first-order formula <j)(x,y) such that the set 



is finite and non-empty (that is, for every imaginary, we can choose finitely many 
tuples to represent it). The following lemma is folklore but I have not been able to 
find a suitable reference. I am grateful to Martin Hils for explaining it to me. 

Lemma 5.1. Suppose that X is an u>- categorical structure that admits weak elimi- 
nation of imaginaries. Then for every open subgroup V < G, there exists a unique 
finite, algebraically closed substructure A C X such that G(a) < V < Ga and 
Ga = ComniG(V). In particular, every commensurator is of the form Ga for 
some A. 

Proof. By the preceding discussion, there exists an imaginary a such that V = G a . 
Let 4> be a formula such that the set D{cf), a) defined in (15.11) is finite and non- 
empty. Let A be the algebraic closure of D(cj>,a). Then clearly G(a) < V. Also, 
from the definition of D((f>,a), V ■ D((f>,a) — D((j>,a) and thus V ■ A = A, so 
that V < Ga- The group G(a) has finite index in Ga because it is equal to the 
kernel of the homomorphism Ga — > Aut(A) given by restriction. To prove that 
Ga = Commc(y), it suffices to check that Ga has no proper supergroups in which 
it is of finite index. To see this, note that, by the definition of algebraically closed, if 
g ■ A =^ A, then the orbit Ga ■ (g • A) is infinite, showing that Ga has infinite index 
in (Ga,.<?)- The uniqueness of A follows from the fact that the commensurator 
of V is uniquely defined and two different algebraically closed substructures have 
different stabilizers. □ 

In fact, the converse of Lemma 15.11 also holds (see [Hi Exercise 7.3.16]) and for 
the examples we consider below, the conclusion of the lemma can easily be verified 
directly. 

If the structure X is moreover homogeneous (as defined in Section [2J , then for 
every finite substructure A C X, the canonical homomorphism Ga — > Aut(A) 
(that is, the restriction to A) is surjective and we have the following. 

Corollary 5.2. Let's, be an u>- categorical, homogeneous structure that admits weak 
elimination of imaginaries and G — Aut(X). Then the following is a complete list 
of the irreducible representations of G: 

{IndQ A (cr) : A C X is finite, algebraically closed and 




(5.1) 



D{4>, Q) = {ceX":tt = {ie X 



111 



■ <j>{x, c)}} 



er is an irreducible representation of Aut(A)}. 



Moreover, this list is without repetitions (only one substructure appears of each 
isomorphism type). 
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Proof. Every representation in the above list is irreducible by Proposition ^. II Con- 
versely, let 7T be an irreducible representation of G. By Theorem 14. 2\ there exist 
open subgroups V < H < G such that H = ComniG(V r ) and an irreducible repre- 
sentation a of H/V such that n = Ind^(cr). By Lemma T5. 11 there exists a finite, 
algebraically closed substructure A C X such that G(a) < V and Ga = H. As the 
quotient map Ga — > Ga/V factors through Ga/G(a) — Aut(A), we can consider 
a as a representation of Aut(A). Finally, that the list is without repetitions follows 
from Proposition 14.11 and homogeneity (the groups Ga and Gb are conjugate iff A 
and B are isomorphic). □ 

It is well known and not difficult to check that Examples (0)-(jvj) from the intro- 
duction admit weak elimination of imaginaries (see (HI Section 4.2] for a general 
method to verify this), so in particular all of their irreducible representations are 
obtained by induction from representations of automorphism groups of finite sub- 
structures. In the case of Soq, we obtain the theorem of Lieberman [L|. The finite 
groups that appear in the representations of and Homeo(2 N ) are the symmet- 
ric groups, while the ones associated to GL(oo,F 9 ) are GL(n,F„). As any finite 
group can be realized as the automorphism group of a finite graph, we see that the 
representations of Aut(R) encode the representations of all finite groups. 

Remark 5.3. In [Olj . Olshanski gives a description of the representations of an- 
other completion of the inductive limit limGL(n,F g ) which is different from our 
GL(oo, F„). It is possible to obtain a proof of his result by our methods as follows. 
Let V be the vector space over ¥ q with basis {ei, e2, . . .} and let e[, e' 2 , ... be the el- 
ements in the dual defined by (ei, e'-) = &y. Let V be the (proper) subspace of the 
dual generated by e' 1; e' 2 , ■ ■ ■■ Our structure then consists of the disjoint union of the 
vector spaces V and V together with binary relations for the pairing VxV'^F,. 
One checks that it is homogeneous and that its automorphism group is isomorphic 
to the one considered in |Q1| . The finite substructures of this structure are pairs 
of the form (A, A'), where A is a finite subspace of V and A' is a finite subspace 
of V. 

Remark. The structure in Example (jvi)) does not admit weak elimination of imagi- 
naries. Another standard example of an w-categorical structure that does not elim- 
inate imaginaries is the group T = (Z/4Z) <N ; the cosets of F/2F are imaginaries 
that cannot be eliminated. 

In (Q, <), finite substructures are rigid, so we have the following. 

Corollary 5.4. The irreducible representations o/Aut(Q) are 

Aut(Q) r\ £ 2 (Q [n] ), n e N. 

As Aut(Q) can be densely embedded in Homeo + (R) (the latter being equipped 
with any of the uniform convergence, pointwise convergence, or compact-open 
topologies which coincide on it), we obtain the result of Megrelishvili [M3J men- 
tioned in the introduction. 

Corollary 5.5. The group Homeo + (M) has no non-trivial unitary representations. 
Proof. Let G = Homeo + (K) and let 7r be a unitary representation of G. Let 

H = {geG: g(Q) = Q} 
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(here Q is regarded as a subset of K) and note that H is a continuous homomorphic 
image of Aut(Q). By Corollary 15.41 tt\h is a direct sum ® i ^ 2 (Q'" i '). Define 
G H by gk{x) = x + f/fc. Then converges to the identity in G but assuming 
that Hi ^ for some i and letting 5 a be the vector in £ 2 (Ql n *l) which is 1 on some 
subset A G Ql"*] and everywhere else, we obtain ||7r(<?fc)5A — <^a|| = V2 for all fc, 
a contradiction with the continuity of n. Hence, ni = for all i and tt is trivial on 
-ff. As H is dense in G, ir must be trivial on G, too. □ 

Remark. In |M3| , Megrelishvili proves the stronger result that Homeo + (K) does not 
have any non-trivial representations by isometries on reflexive Banach spaces. 

We finally remark that the continuity assumption in our definition of a represen- 
tation is often not restrictive because of the phenomenon of automatic continuity. 
Say that a Polish group G satisfies the automatic continuity property if every homo- 
morphism / : G — > H to a separable group H is continuous. This quite remarkable 
property has been verified for many non-locally compact groups, including Ex- 
amples (0)-(jvj) from the introduction (for Soo, GL(oo,Fq), and Aut(R) it is due to 
Kechris-Rosendal [KRJ and Hodges-Hodkinson-Lascar-Shelah [HHLS]; for Aut(Q) 
and Homeo(2 N ), it is a result of Rosendal-Solecki |RS| ). 

Corollary 5.6. Let G be an oligomorphic group that satisfies the automatic con- 
tinuity property. Then the conclusion of Theorem \4-2\ applies to any ( not a priori 
continuous) unitary representation of G on a separable Hilbert space. 

Of course, the condition that the Hilbert space be separable is necessary: every 
oligomorphic group considered as discrete has its left-regular representation which 
is not of the kind described in the theorem. 

6. Property (T) 

We recall the definition of property (T) for topological groups. 

Definition 6.1. Let G be a group, Q C G, e > 0. If w is a unitary representation of 
G, we say that a non-zero vector £ € H(tv) is (Q, e)-invariant for n if for all x G Q, 
||7r(x)£ — £|| < e ||£||. The topological group G is said to have Kazhdan's property 
( T) if there exist a compact Q C G and e > such that every representation 7r of 
G that has a (Q, e)-invariant vector, actually has an invariant vector. G has the 
strong property (T) if Q can be chosen to be finite. The set Q is called a Kazhdan 
set for G. 

This property has mostly been studied for locally compact groups, where it has 
found many applications, but by now, there are also some non-locally compact 
examples. Of course, groups that have no unitary representations trivially have 
property (T) but there are also some large groups that have property (T) for non- 
trivial reasons. The first examples of this type were the loop groups over SL(n,C) 
(Shalom [S]), another is the infinite-dimensional unitary group (Bekka [B]). 

We note that property (FH), which is equivalent to (T) for locally compact Polish 
groups (see |BdlHV| ). is strictly weaker in general. While all Roelcke precompact 
groups have property (FH) (Proposition I2.3[) , additional work is needed to find 
appropriate Kazhdan sets. 

As every representation of a Roelcke precompact subgroup of S'oo splits as a 
sum of irreducibles, to verify that such a group has property (T), it suffices to find 
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(Q, e) such that sup g6 g IKCflOC — £|| > 6 f° r every non-trivial, irreducible ir and unit 
vector £ £ W.(n). 

We have seen in the previous section that when an w-categorical structure X ad- 
mits weak elimination of imaginaries, all irreducible representations of G = Aut(X) 
can be extracted from the action G rx X. As we are concerned with non-trivial 
representations, it will be convenient to disregard the points in X that are fixed by 
G. Denote Xo = X \ {a € X : G ■ a = a}. The next proposition shows that to 
verify property (T), it suffices to consider only tensor powers of the representation 
^ 2 (Xo). 

Proposition 6.2. Let X be an uj -categorical structure that admits weak elimination 
of imaginaries and G = Aut(X). Then every non-trivial, irreducible representation 
of G is a subrepresentation o/£ 2 (Xq) for some n > 0. 

Proof. Let n be an irreducible representation of G. In the same way as in the proof 
of Corollarv l5.21 we see that there exists an algebraically closed, finite substructure 
A C X such that ir is equivalent to Ind GA (o") for some irreducible representation 
a of the finite group K — Ga/G(a)- As every irreducible representation of a finite 
group is contained in its left- regular representation, in the same way as in (|4.3I) , we 
obtain: 

tt - Indg» < Indg A (A*) S= £ 2 (G/G {A) ). 

Hk= | A |, then G/G ( A ) is an orbit of the action G rx X fe , so ^ 2 (G/G(a)) C £ 2 (X k ). 
To finish the proof, we check by induction on k that £ 2 (X. k ) splits as a direct sum 
0" =1 ^(X™ 1 ). If k = 0, we can take s = 1 and n x = 0. Denote Y = X \ X and 
observe that as the action G rx X has only finitely many orbits, Y is finite. Now 
we deduce the statement for k + 1 from the one for k: 

s 

P{X k+1 )=l 2 {X)®f(X k ) = (£ 2 {X )®£ 2 (Y)) ® (®^(XJ*)) 

»=i 

s s 

= ®^ 2 (x^ +1 )®|y|.®£ 2 (x^). 

i=l i=l 

Finally, as 7r is irreducible and non-trivial, it is a subrepresentation of one of the 
£ 2 (X£*) for some m > 0. □ 

If y is a set, denote by £\(Y) the subset of ^(Y) consisting of all non-negative 
£ x functions of norm 1. 

Lemma 6.3. Suppose that a group G acts on a set Y so that there exists a subset 
Q C G and e > such that for every f 6 £^_(Y), 

(6.1) sup /-/Hi >e- 

geQ 

Then (|6.ip ZioZrfs aZso /or ewery / £ £^_(Y n ) (with the diagonal action G rx Y n ) for 
every n > 2. 
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Proof. The proof is by induction on n. Let / € i\{Y n+1 ). Define / G l\(Y n ) by 
/(z) = J2 y eY /( s > Then for an y 5 € G, 



^El/V 1 -*) -/(«)! 

x y y 

< EE • • y) - /&»)! = lis • / - /Hi . 



showing that (|6.f [1 holds for / if it holds for /. 



□ 



Proposition 6.4. Suppose that X is an uj- categorical structure that admits weak 
elimination of imaginaries and G — Aut(X). Then if Q C G is compact and 

(6.2) inf su P || 5 -/-/|| 1 >0, 

/e^(x ) 9G Q 

Q is a Kazhdan set for G. Conversely, if the action G rv Xo has only infinite orbits 
and Q is a Kazhdan set for G, then (|6.2p holds. 



Proof. Suppose first that (|6.2[) holds and set e = inf^i ( Xo ) sup sg Q \\g ■ f — f\\ v 



By Proposition 16.21 to see that Q is a Kazhdan set for G, it suffices to show that 
for every n > and every / G ^(XJ), sup geQ \\g ■ f - f\\ 2 > (e/2) ||/|| 2 . Suppose 
that / G £ 2 (X%), ||/|| 2 = 1, g G G. Set /» = |/(x)| 2 . Then / G ^(X^) and we 
have 



9-f-f 



t = E ll/Gr 1 -*)! 2 -!/®! 2 ! 

= E |l/(^ 1 -«)|-|/(s)l|-(l/(ff- 1 -S)| + 1/(5)1) 
sex;,* 

= <|«7- l/l + 1/1) 

< ||.g-/-/|| 2 ||.g-|/l + l/lll 2 <2||.g./-/|| 2 . 



Combining this with Lemma 16.31 finishes the proof. 

Now suppose that (|6.2[) does not hold but there exists e > such that (Q, e) is a 
Kazhdan pair for G. Then there exists / G ^(Xq) such that sup sg Q ||<7 • / — f\\^ < 



e . Using the inequality \a — b\ < 



b |, which holds for all non-negative real 



numbers a and b, we see that f 1 / 2 is a (Q, e)-invariant vector for the representation 
£ 2 (Xo). By property (T), £ 2 (Xo) has an invariant vector /o and then {a G Xo : 
|/o(a)| = max|/o|} is a finite G-invariant set in Xo, which is a contradiction with 
the hypothesis that the action G rx Xo has infinite orbits. □ 

The next lemma shows that at least for certain well-behaved structures, we 
can always construct a compact set Q C G that satisfies (|6.2I) . Recall that an 
w-categorical structure has no algebraicity if the algebraic closure of every finite 
substructure A is A itself. This is equivalent to the condition that the stabilizer 
G(a) nas infinite orbits on X \ A. 
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Lemma 6.5. Let X be a homogeneous, relational structure with no algebraicity 
and let G = Aut(X). Then there exists a compact set Q C G such that for every 
/G^(X), 

sup \\g- f-fW, > 1/2. 
geQ 

Proof. We will use a back-and- forth construction. Enumerate X = {di, a.2, . . .} and 
set A„ = {ai : i < n} for n > 0. We will define inductively finite families Si, S2, ■ ■ ■ 
of finite partial automorphisms of X with the following properties: 

(1) Si = {0}; 

(2) if <f> £ Sn+ii there exists a unique 4>' £ S n such that </> D </>'; 

(3) for every </> € <S2n, A n C dom.0 and for every <j> £ San+i, A„ C ran0; 

(4) for every £ S2n U San+ii dom0 n ran</> C A„; 

(5) for every <j> £ San— 1> there exist ipi,ip2, ■ ■■ , tp2 n + 1 € San such that tpi D <j) 
for every i and ran D ran -0j = ran for all i ^ j; 

(6) for every <j> £ S^n, there exist ipi, ip2, ■ • ■ , 4'2 n + 1 € S2n+i such that ^ 3 (i 
for every i and dom^j (~l dom^j = dom^ for all i =/= j. 

One can view the sets S n as the levels of a finitely splitting rooted tree. 

Recall that for a homogeneous structure the no algebraicity assumption can be 
reformulated as the following extension property (see Hodges [H ): 

for every pair of finite structures A,B, embeddings ip: A — > X 
and 0: A — » B, and finite set D C X, there exists an embedding 
0: B — > X such that o = ijj and 0(B) nDC ^(A). 
Start the construction with Si = {0}. Suppose now that S^n—i has been con- 
structed and proceed to build S^n. For every <p £ S^n-i, we will construct a set E$ 
of extensions of 4> to put in San- If a n G dom0, set 25^ = {(/>}. If a„ ^ dom^>, let 
B = dom</> U {a„} and construct -E^ = {ipi, ifa,. .., ip2 n + 1 } inductively as follows. 
Suppose ipi , . . . , ipk have been constructed and apply the extension property to the 
inclusion map dom <p — > B and the embedding <f> '■ dom <f> — > X to obtain a partial 
automorphism tpk+i with domain B that extends (j) and such that 

(6.3) rani/>/c+i H ( [J ran^ U C ran^. 

Finally, set 6*2,1 = U0es 2 1 ^° cons truct Sa n +i apply the same procedure 
symmetrically (replacing <j> with </> _1 ). Conditions (|3|. |5]), and the existence 
part of ([2]) are satisfied by construction. For the uniqueness part of ([2]), note that 
any two distinct elements of S n are incomparable. Finally, suppose that condition 
([1)1 is not verified and let i be the least natural number such that there exists 
4>i £ Si and a £ (dom^ Hran 4>i)\ A^/2j ■ We will consider the case when i — 2n is 
even, the other one being similar. Denote by </>2n-i the element in San-i such that 
02n-i Q (j) 2n . By (|6.3| . ran02n H dom0 2n C ran0 2n -i, so a £ ran0 2 n-i- On the 
other hand, by construction, dom02n = dom02n-i U {a n }. Hence a £ dom </>2n-i, 
a contradiction with the minimality of i. 

Now {S n } n forms naturally an inverse system with the maps 

S n +1 S n , 4> H- the unique (f>' £ S n such that <fi C 0. 

Denote by Q the inverse limit, i.e. 

Q = {g £ G : Vn30 G 5„ <j> C .g}. 



22 



TODOR TSANKOV 



(Every map in the inverse limit is a full automorphism of X because of condition 
(HUO As the sets S n are finite, Q is compact. 

We now check that for any / g ^(X), there exists g £ Q such that ||g • / — /(^ > 
1/2. We build inductively a sequence fa,<f>2, • ■ ■ such that ^ S Si and fa C . 
Let <j)\ = 0. Suppose that fa n -i is already chosen. If a n € dom</>2 n _i, set fan = 
fan-i- If in ^ dom(/)2n-i, by ([5]) and the fact that H/l^ = 1, there exists fa n G £2™ 
such that 

(6.4) /(0 2n (a„)) <2-(" +1 >. 

Now we proceed to choose fa n +i- If d« ^ dom02n-ij let fa n +\ € <S2n+i be an 
arbitrary extension of 02n- If a„ G dom02n-i, then by (|4|), a„ ^ ran^2n-i, so 
a n ^ vanfan either (as in this case, (f>2n = fan-i)- Now, using ([6]), we can choose 

fan+1 € S^n+l SO that 

(6-5) /(fe 1 +iK))<2- ( " +1) . 

In summary, we obtained that for every n at least one of the inequalities (|6 .4[) and 
(|6.5p holds. Let g — \J. fa. Combining (|6.4|) and (|6.5p . we have that for every n, 

min(/(a„),/( 5 - 1 -a„)) <2-(™+ 1 ). 
Now a calculation yields: 

oo 

lls-/-/lli = E l/Or 1 -0-/MI 



n=l 
oo 

£ 

n=l 



max (/(a„), f(g 1 • o„)) - min (/(«„), /(.g 1 • a„)) 



> n/iii -E min (/( a ")'/^ 1,a «)) 

n=l 



> l-^2-(" +1 ) = 1/2, 
n=l 

finishing the proof. □ 

Combining everything we have so far, we obtain the following. 

Theorem 6.6. Let X be an oj- categorical, relational, homogeneous structure with 
no algebraicity that admits weak elimination of imaginaries. Then Aut(X) has 
property (T). 



Proof. Follows from Proposition 16.41 and Lemma 16.51 □ 

As the next theorem shows, in many concrete examples, it is not difficult to find 
finite Kazhdan sets. In fact, I do not know whether, in the setting of Theorem l6.6[ 
it is always possible to do so (cf. Question @). 

Theorem 6.7. All of the following groups have a Kazhdan set with two elements: 
Soo, Aut(Q), GL(oo,F 9 ), Homeo(2 N ), Aut(R). 



Proof. In view of Proposition 16. 4[ to prove that the automorphism group G of 
an w-categorical structure X with weak elimination of imaginaries admits a finite 
Kazhdan set, it suffices to find a finitely-generated subgroup T < G such that the 
action T rx Xq is non-amenable (i.e. (|6.2| holds with Q a generating set for T). In 
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fact, for all of the above groups we will find a copy of the free group F2 in G that 
acts freely on Xo. As it is well-known that any free action of F2 is non-amenable, 
this will complete the proof. 

^oo . Consider the left action of F2 on itself. This gives a copy of F2 in the group 
of all permutations of F2 that acts freely. 

Aut(Q). It suffices to find an ordering on the group F2 which is left-invariant 
and isomorphic to Q. Then the left action of F2 on itself will produce the required 
embedding F2 <—> Aut(Q). It is well known that F2 admits a bi-invariant (invariant 
under multiplication on both sides) linear ordering (see, for example, [RlJ). We 
now check that any bi-invariant ordering on F2 is dense without endpoints. First, 
if a; > 1, then x 2 > x and if cc < 1, x 2 < x, showing that the ordering has no 
endpoints. To see that it is dense, it suffices, for every x > 1, to find z such that 
1 < z < x. By bi-invariance, all conjugates of x are > 1. Let now y be an arbitrary 
element that does not commute with x. If x y < x, we are done and if x v > x, by 
conjugating with y~ 1 , we obtain that x y < x. 

GL(oo,F 9 ). Label a basis of the vector space by the elements of F2 and let F2 
act freely on this basis in the natural way. This action extends to an action on 
the whole vector space. As F2 is torsion-free, no finite, non-empty subset of F2 is 
invariant under any non-identity element of the group, so the support of every non- 
zero vector is moved by every non-trivial element of Fa, showing that the action is 
free (when restricted to the non-zero elements of the vector space). 

Homco(2 N ). Consider the natural shift action F2 rv 2 F2 given by (g ■ oS){h) = 
^(g^h). Since this action is by homeomorphisms, it gives an action F2 rx Clopen(2 F2 ). 
Using a similar argument to the one above, we see that its restriction to Clopen(2 F2 )\ 
{0,2 Fa } is free. 

Aut(R). Consider a random right Cayley graph of the free group constructed in 
the following way. Let S be a random set of unordered pairs {g, <7 -1 } of elements 
of F2 \ {1}, i.e. each pair is included or not in S independently with probability 
1/2. The vertices of the graph are the elements of F2 and two vertices x, y £ F2 are 
connected by an edge iff x~ x y £ S. By Cameron |C2| . the random Cayley graph 
of F2 is isomorphic to R with probability 1, so in particular, Cayley graphs of F2 
isomorphic to R exist. As F2 acts freely by isomorphisms on any of its Cayley 
graphs, we obtain the desired embedding F2 °-> Aut(R). □ 

Remark. It is also possible, using the method of Bekka [BJ, to find optimal Kazhdan 
constants for those Kazhdan sets. In fact, the constant is the same as in [B] . 

Remark. We note that not all oligomorphic groups have the strong property (T). 
Indeed, the group in Example (jvlj admits (Z/2Z) N as a quotient and the latter does 
not admit a finite Kazhdan set by (Bj Proposition 5] (this can also easily be seen 
directly) . 

Remark. As was already noted in [Bj . there is no special connection between 
property (T) and amenability for non-locally compact groups. Of the groups in 
Theorem 16. 7\ all are amenable except Homeo(2 N ). (A topological group is called 
amenable if every time it acts continuously on a compact space, there is an invariant 
measure.) 

Remark. It is an open problem whether there exists a discrete subgroup of Aut(Q) 
with property (T). (A discrete group embeds in Aut(Q) iff it acts faithfully by 
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orientation-preserving homeomorphisms on the reals iff it is left-orderable; see Mor- 
ris |M4| ) . I am grateful to the referee for pointing this out. 

We conclude with two open problems. 
Questions: 

(1) Does every closed oligomorphic subgroup of Soo have property (T)? More 
generally, does every Roelcke precompact Polish group have property (T)? 

(2) Suppose that X is an w-categorical, relational, homogeneous structure with 
no algebraicity. Is it always the case that the action Aut(X) r\ X is non- 
amenable, i.e. must there always exist a finite set Q C Aut(X) such that 
for every / <E ^+(X), 

sup /Hi >0? 

g&Q 
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